Methods and systems for automatic digital image enhancement with local adjustment

ABSTRACT

Embodiments of the present invention comprise systems, methods and devices for digital image contrast enhancement.

FIELD OF THE INVENTION

Embodiments of the present invention comprise methods and systems forautomatically enhancing the contrast and other attributes of digitalimages.

BACKGROUND OF THE INVENTION

The quality of a digital image may be affected by the light conditionsat the time of image capture, the quality of the capture device and theskill of the person capturing the image. As a result of these and otherfactors, digital images often suffer from poor contrast and otherproblems in one or more areas of the image. In some cases, the lightestarea of the image is “washed-out” from over-exposure, incorrect capturedevice settings or other reasons. In other cases, a dark area of theimage is “blacked-out” from under-exposure, incorrect capture devicesettings or other reasons. In these and other cases, portions of theimage suffer from poor contrast as some image values are not perceptibleto the human observer.

Some manual methods exist for adjusting the level, brightness, contrastor other attributes of an image to improve these types of problems.However, these methods typically require a user to select the propertool and demonstrate some degree of skill in applying the tool to animage. Often, image contrast in one area is improved only at a cost ofimage degradation in another area.

Some automated methods for image contrast enhancement also exist.Histogram equalization is a popular method for contrast enhancementbecause of its effectiveness and relative simplicity. Histogramequalization may be classified into two groups based on thetransformation function used: Global histogram equalization uses aglobal transformation function for the entire image. Global methods arefast and simple, but their contrast-enhancement power is relatively low.Global methods cannot adapt to local brightness features of an imagebecause only global histogram information is used over the image. Localhistogram equalization generally can enhance overall contrasteffectively, but the complexity of computation is high due to the use offully overlapped sub-blocks.

One method of local histogram equalization uses block-overlappedhistogram equalization. In this method, a rectangular sub-block of aninput image is selected and a histogram of that sub-block is determined.A histogram-equalization f unction is then calculated for thatsub-block. The center pixel of the sub-block is then moved, therebydefining a new sub-block, and the computation is performed again with anew histogram and equalization function. The procedure is repeated foreach pixel in the image. Obviously, the computation complexity is veryhigh relative to a global method.

BRIEF SUMMARY OF THE INVENTION

Embodiments of the present invention comprise systems, methods anddevices for automatically enhancing contrast and other characteristicsof an image.

07 Some embodiments of the present invention comprise determination ofan image histogram, classification of the image based on histogramcharacteristics and enhancement of the image withclassification-specific methods.

The foregoing and other objectives, features, and advantages of theinvention will be more readily understood upon consideration of thefollowing detailed description of the invention taken in conjunctionwith the accompanying drawings.

BRIEF DESCRIPTION OF THE SEVERAL DRAWINGS

FIG. 1 is a diagram showing image histograms corresponding to exemplaryimage types;

FIG. 2 is a flow chart showing exemplary image classification methods;

FIG. 3 is a chart showing an exemplary gamma correction curve;

FIG. 4 shows exemplary pseudo code for generating a global luminancemap;

FIG. 5 is an exemplary luminance map for a Type B image;

FIG. 6 is an exemplary luminance map for a Type C image;

FIG. 7 is an exemplary luminance map for a Type D image;

FIG. 8 shows exemplary pseudo code for generating a local luminance map;and

FIG. 9 is a diagram showing an exemplary luminance map averaging method.

DETAILED DESCRIPTION

Embodiments of the present invention will be best understood byreference to the drawings, wherein like parts are designated by likenumerals throughout. The figures listed above are expressly incorporatedas part of this detailed description.

It will be readily understood that the components of the presentinvention, as generally described and illustrated in the figures herein,could be arranged and designed in a wide variety of differentconfigurations. Thus, the following more detailed description of theembodiments of the methods and systems of the present invention is notintended to limit the scope of the invention but it is merelyrepresentative of the presently preferred embodiments of the invention.

Elements of embodiments of the present invention may be embodied inhardware, firmware and/or software. While exemplary embodiments revealedherein may only describe one of these forms, it is to be understood thatone skilled in the art would be able to effectuate these elements in anyof these forms while resting within the scope of the present invention.

Some embodiments of the present invention comprise a digital imageenhancement method that is capable of enhancing dark images or darkportions of an image. In these embodiments a histogram analysis isperformed on the entire image. The global histogram is then classifiedas one of a fixed set of image/histogram types. A global mappingalgorithm may then be calculated for the luminance channel. This may bea piecewise linear mapping. Pivot points in the piecewise linear map maybe determined by a process which is a combination of histogramequalization and gamma correction. Linear mapping between pivot pointsmakes the map easy to calculate. Look-up tables may be easily generatedon-the-fly making the method suitable for digital cameras and otherdevices with limited processing power. In some embodiments, no change ismade to the chrominance channels during enhancement.

Some embodiments of the present invention comprise a characterizationmethod comprising performing a histogram analysis of the luminancechannel of an image and characterizing or classifying the imageaccording to histogram characteristics. Empirical thresholds may be usedto classify the image. In some embodiments, images may be classified asone of a finite set of image types. In an exemplary embodiment, shown inFIGS. 1A-1D, four types are used.

Exemplary image type A, shown in FIG. 1A, comprises a sufficient numberof “bright” pixels, but not many “dark” pixels. These are considered“normal” images and adjustment of these images may be omitted.

Exemplary image type B, shown in FIG. 1B, comprises a high number of“bright” pixels and a high number of “dark” pixels with fewer pixels inthe midrange. Images with this type of polarized pattern may benefitfrom an image enhancement that enhances the dark pixels, but leaves thebrighter pixels unmodified. This type of image may occur when the mainobject of an image is dark while the background is very bright.

Exemplary image type C, shown in FIG. 1C, comprises a low number of“bright” pixels and a high number of “dark” pixels. Images with thistype of “predominantly dark” pattern may benefit from an imageenhancement that stretches the dark pixels levels and compresses theother pixel levels.

Exemplary image type D, shown in FIG. 1D, comprises a high number ofmid-range pixels and a relatively low number of dark and bright pixels.Images with this type of “mid-range” pattern may benefit from an imageenhancement that increases the brightness of mid-range pixels, stretchesthe very dark pixel levels and compresses the very bright pixel levels.A brightness offset may be determined by accumulating pixels from thehigh end of the histogram until reaching a specified small percentage ofpixels. A difference between the luminance value reached when thispercentage has accumulated and the max luminance value, e.g. 255 for8-bit images, or part of this difference, can be used as a brightnessoffset. The brightness offset may be used to generate a luminancemapping.

In some embodiments of the present invention, an image may be classifiedas a specific image type according to the chart shown in FIG. 2. Inthese embodiments, image histogram data is analyzed. In this analysis, afirst determination 21 is made to determine if a first condition is metwherein a sufficient number of bright pixels exist and an insufficientnumber of dark pixels exist. If this first condition is met, the imageis classified as a “Type A” image 22.

If this first condition is not met, a second determination 23 may bemade to determine if a second condition is met wherein many very brightpixels are present in the image. If this condition is met, the image isclassified as a “Type B” image 24.

If the second condition is not met, a third determination 25 may be madeto determine if a third condition is met wherein many very dark pixelsare present in the image. If this condition is met, the image isclassified as a “Type C” image 26. If this condition is not met, alongwith the first and second conditions, the image is classified as a “TypeD” image 27.

In some exemplary embodiments of the present invention, a luminancehistogram may comprise eight equal-sized bins. In other exemplaryembodiments, a luminance histogram used to characterize or classify animage may comprise a number of bins between 4 and 20.

In some of these embodiments, a sufficient number of bright pixels, asdetermined for the first determination 21, may exist when the top ⅜ ofthe bins of the histogram hold more than 40% of the pixels. In otherexemplary embodiments, this condition may be met when the top ⅛ to ½ ofthe bins holds more than 30% to 50% of the total pixels in the image.

In some of these embodiments, an insufficient number of dark pixels, asdetermined for the first determination 21, may exist when two conditionsare met: a) the percentage of the bottom ⅛ bin is less than 10% of thetotal number of pixels and b) the percentage of the bottom ⅛ to ⅜ of thebins is less than 20% of the total number of pixels. In other exemplaryembodiments, this condition may be met when the bottom 1/16 to ¼ binssimply contains less than 5% to 15% of the image pixels. In otherexemplary embodiments, this condition may be met when the bottom ⅛ to ⅜of the bins simply contain less than 10% to 30% of the image pixels. Inother exemplary embodiments, this condition may be met when twoconditions are met: a) the percentage of the bottom 1/16 to ¼ bins isless than 5% to 15% of the total number of pixels and b) the percentageof the bottom ⅛ to ⅜ bins is less than 10% to 30% of the total number ofpixels.

In some of these embodiments, the condition of many very bright pixelsbeing present, as determined for the second determination 23, may existwhen two conditions are met: a) the percentage of the top bin is notless than 6.25% and b) the percentage of the top two bins is not lessthan 12.5%. In other exemplary embodiments, this condition may be metwhen the top 1/16 to ¼ of the bins simply contains not less than 2.5% to12.5% of the image pixels. In other exemplary embodiments, thiscondition may be met when the top ⅛ to ⅜ of the bins simply contain notless than 6% to 18% of the image pixels. In other exemplary embodiments,this condition may be met when two conditions are met: a) the percentageof the top 1/16 to ¼ of the bins is not less than 2.5% to 12.5% of thetotal number of pixels and b) the percentage of the top ⅛ to ⅜ of thebins is not less than 6% to 18% of the total number of pixels.

In some of these embodiments, the condition of many very dark pixelsbeing present, as determined for the third determination 25, may existwhen a condition is met wherein the percentage of the bottom two bins isnot less than 20%. In other exemplary embodiments, this condition may bemet when the bottom ⅛ to ⅜ of the bins simply contain not less than 10%to 30% of the image pixels.

Other embodiments of the present invention may comprise othercombinations of histogram bin values and pixel percentages outside thespecific numeric values and numeric ranges described above, but withinthe textual description of the conditions.

Global Luminance Mapping

Many luminance mapping methods may be used in embodiments of the presentinvention. In some exemplary embodiments, mapping may be done in a YUVcolor space. In some of these exemplary embodiments, mapping may beperformed only on the luminance, Y, channel.

In some embodiments, luminance mapping may be performed using apiecewise linear map comprising a plurality of continuous linearrelationships. These linear relationships may be represented by a fewpivot points. With piecewise linear mapping, one can easily calculate apixel level look-up table on the fly. In embodiments of the presentinvention, pivot points may be spaced along a vertical (output) axis.Since the output is histogram equalized, spacing along this axis isusually well behaved, while spacing along the input axis may havecollapsed pivot points if there are no pixel values falling between twopivot points.

In some embodiments, the start point of a first linear relationship canbe fixed at the origin, at point (0,0). In some embodiments, the endpoint of the last linear relationship is a maximum value point (e.g.,255,255 for an 8-bit image). Other, adjustable pivot points in thepiecewise linear relationships may be chosen to ensure that all linesmonotonically increase. These points may be determined based on theprinciple of histogram equalization or by other methods. Theintermediate or adjustable pivot points may be spaced out along thevertical (output) axis of the mapping curve. In some embodiments, atarget percentage of total pixels may be specified to determine eachpivot point. The highest pivot point may be referred to as the “brightpivot point.” Pixels above the bright pivot point may be classified as“bright pixels.”

In an exemplary luminance map embodiment, 4 pivot points may be selectedin addition to the start point at (0,0), and the end point at (255,255).Targeted percentages for the pivot points may be selected as 10%, 25%,50% and 75% or other values that are spread throughout the percentagerange. In this case, the point corresponding to 75% may be consideredthe “bright pivot point.” In other embodiments, other percentage valuesmay be specified.

The Y-coordinate of a pivot point, denoted as mapped_Y[i] (i.e., themapped luminance value), can be then straightforwardly calculated bymapped_(—) Y[i]=Y max*target_percent[i ],   (1)where i is the pivot-point index and target_percent[i] denotes thetarget percentage for the pivot point i.

Based on the principle of histogram equalization, the X-coordinate of apivot point should be the least luminance value at which the accumulatedhistogram reaches its target percentage. Hence, to calculate theX-coordinate of a pivot point, we first calculate the target count ofthe pivot point, denoted as target_count[i]:target_count[i=target_percent[i]*total_pixels,   (2)where total_pixels denotes the total number of pixels.

The X-coordinate of the pivot point i, denoted as actual_Y[i] (i.e., theactual or original luminance value), can be determined as follows:

actual_Y[i]=the minimal k such that $\begin{matrix}{{{\sum\limits_{j = 0}^{k}{{histogram}\quad\lbrack j\rbrack}} > {{target\_ count}\quad\lbrack i\rbrack}},} & (3)\end{matrix}$where histogram [j] is the number of pixels with value j.

Thus, the coordinates of the pivot point i can be denoted as (actual_Y[i], mapped_Y[i]). To avoid mapping lines going up too fast and to betterpreserve the natural look of the images, the slopes of the mapping linescan be regulated by selected maximum slopes. To facilitate fastconversion of luminance values, a mapping look-up table can becalculated by linear interpolation between adjacent pivot points.

Global Luminance Mapping with Gamma Adjustment

In some embodiments of the present invention, a different luminance mapmay be generated for some of the image types identified in the imageclassification processes described above or for image types classifiedby other methods.

In some embodiments, for a Type B image, with many very bright pixels,the levels above the bright pivot point may be preserved. This may beachieved by setting the target count for a pivot point equal to thedifference between the total number of pixels and the number of brightpixels multiplied by the target percentage for the pivot point dividedby a quantity equal to the bright pivot point percentage. Thismodification will tend to preserve the levels above the bright pivotpoint. In these embodiments, the target count for a pivot point, for aType C or Type D image, may be calculated as the target percentage forthe pivot point multiplied by the total number of pixels.

Once the initial pivot points have been determined, adjusted, mappedpivot points may be calculated by applying a gamma adjustment curve. Anexemplary gamma adjustment curve may be generated using the followingequations: $\begin{matrix}{{{gamma\_ tab}\quad\lbrack Y\rbrack} = {{bright\_ threshold}*\left( \frac{Y}{bright\_ threshold} \right)^{gamma}}} \\{{{if}\quad 0} \leq Y < {bright\_ threshold}} \\{{{gamma\_ tab}\quad\lbrack Y\rbrack} = {{Y\quad{if}\quad{bright\_ threshold}} \leq Y \leq 255}}\end{matrix}$

Using these exemplary equations, when gamma is less than 1, the gammaadjustment will brighten pixels below the bright pivot point andpreserve those above that point. In some embodiments, good results areobtained with a gamma value of 0.6 and a bright pivot point at 192. Anexemplary gamma adjustment curve generated with these values is shown inFIG. 3.

Once the gamma adjustment curve is generated, it may be applied to aluminance map using a weighting factor. Adjusted, mapped luminancevalues may be obtained by a weighted average of the luminance mappedvalues and the gamma curve values. The pseudo code example, depicted inFIG. 4, uses a weighting value, w, that may range from 0 to 1.

In the exemplary method depicted in FIG. 4, wherein pseudo code 40 isused to describe these embodiments, some optional steps are included. Afirst optional step 41 is used to ensure that the mapped luminance valuedoes not fall below the original input value thereby maintaining overallimage brightness. A second optional step 42 is also shown wherein abrightness offset is used to preserve higher luminance values in aspecific type of image (Type D). Another optional step 43 is used toapply a weighting factor and adjust the relative weight of the initialluminance map and the gamma curve in the adjusted map values. Yetanother optional step 44 may be used to clip adjusted map values to theproper range. In some embodiments, other methods may be used to forceadjusted values that have been adjusted out of range into the properrange.

In some embodiments of the present invention, a further restriction maybe placed on the adjusted values. In these embodiments, a maximum slopelimit may be applied to ensure that the map's linear segments do notincrease too quickly. A different slope limitation may be applied toeach segment of the piecewise linear relationship. Pivot points may beadjusted downward until this limitation is satisfied.

After selected adjustments, described above, have been applied and finalpivot points have been calculated, a pixel level mapping look-up tablemay be generated by linearly interpolating between adjacent pivotpoints.

An exemplary final luminance map for a typical Type B image is shown inFIG. 5. An exemplary final luminance map for a typical Type C image isshown in FIG. 6. An exemplary final luminance map for a typical Type Dimage is shown in FIG. 7.

Local Adjustment/Enhancement

Some embodiments of the present invention may comprise localenhancement, wherein a local block mapping curve for each non-overlappedblock may be calculated in a way similar to the global enhancementmethods described above, but using a local or block histogram, insteadof the global histogram, to calculate the pivot points.

In some local enhancement embodiments, an image may be divided into manynon-overlapped blocks. For example, for a 2M-pixel image with width 1632pixel and height 1224 pixel, the image may be divided into a 16×12 gridof blocks. Each block's size is 102 by 102 pixels. In a one-passhistogram analysis, both global and local histograms may be obtained.

In some embodiments, local adjustment curves may be determined. Localluminance mapping may be determined in a way similarly to the globalmapping described above. In some embodiments, it may be performed in theYUV color space, and may be performed only on the luminance channel—Y.For each block, the mapping curve may comprise several continuouspiecewise linear lines, which may be referred to as a block mappingcurve and which can be summarized by a few pivot points. Typically, thestarting point (0,0) and the ending point (Ymax, Ymax) are included asthe first and the last pivot points, where Ymax is the maximum value forluminance signal, e.g., Ymax=255 for 8-bit images. The other pivotpoints are chosen to make sure all lines monotonically increase. Atarget percentage of total pixels may be specified to determine eachpivot point. For example, not including (0,0) and (Ymax, Ymax), we maychoose 4 pivot points with targeted percentages of 10%, 25%, 50% and 75%respectively. The pivot points may be determined by a histogramequalization-like procedure similar to those described above in relationto global methods. The pseudo code for an exemplary embodiment fordetermining the coordinates of the pivot points is shown in FIG. 8.These embodiments comprise a slightly simpler method than thosedescribed in relation to global enhancement, however the same methodsused in the described global embodiments may be applied to localenhancement.

In some embodiments, the local mapped luminance value may be obtained byinterpolation. Due to the difference among local mapping curves ofneighboring blocks, if all pixels inside a block are mapped by onlyusing this block's own local mapping curve, blocking artifacts wouldoften occur at block boundaries. To avoid blocking artifacts, the localmapped luminance value of a pixel may be calculated by interpolating themapping curves of its four nearby blocks.

An exemplary interpolation method is depicted in FIG. 9. In theseembodiments, pixel A 90 has four nearby blocks 91-94. We first obtainthe four mapped values of its original luminance value by the blockmapping curves of those four nearby blocks. We denote these four mappedvalues as Y_ul, Y_ur, Y_bl, and Y_br for the upper left, the upperright, the bottom left and the bottom right blocks respectively. Let xand y denote, respectively, the horizontal and vertical distances fromthe pixel A to the center of its block, which is the bottom right blockin this case. Let w and h denote the width and height of a block,respectively. Use Y_u and Y_b to denote the horizontally interpolatedvalues of the upper two blocks and the lower two blocks. Then the finalmapped value of the pixel A for local enhancement, denoted as local_Y,can be calculated by a weighted sum of Y_ul, Y_ur, Y_bl, and Y_br withweights inversely proportional to distances, namely:Y _(—) u=Y _(—) ul*x/w+Y _(—) ur*(1−x/w)Y _(—) b=Y _(—) bl*x/w+Y _(—) br*(1−x/w)local_(—) Y=Y _(—) u*y/h+Y _(—) b*(1−y/h)   (4)

The effect of the interpolation of Y_ul, Y_ur, Y_bl, and Y_br isequivalent to using an interpolation of four nearby block mapping curvesto map the original luminance value of the pixel A. This mapping-curveinterpolation effectively avoids any blocking artifacts. Hence node-blocking filter is needed and the blur usually caused by thede-blocking filter can also be avoided.

Combined Global and Local Enhancement

Some embodiments of the present invention may comprise a localadjustment skipping method. Compared to global adjustment, localadjustment is relatively expensive in terms of complexity. To lower thecomplexity, local adjustment can be skipped when it is not necessary.The local-adjustment skip may be done on both the picture level and theblock level. In order to determine the necessity of local adjustments, alocal property about the darkness of each block may be acquire. The term“dark block” may refer to a block in which the majority of pixels aredarker than a threshold value.

For some picture-level local-adjustment skipping embodiment, the localadjustment for entire image may be skipped when the picture has a totalnumber of dark blocks greater than a specified threshold. For this kindof image, since there are a lot of dark blocks, the dark blockhistograms would be sufficiently represented in the histogram of theglobal image. Hence, global adjustment alone would be sufficient and theexpensive local adjustments can be skipped.

For block-level, local-adjustment skipping embodiments, the localadjustment may be applied only to the dark blocks and the pixels nearthe dark blocks. Local adjustment curves may be obtained for dark blocksonly, and the interpolations are performed for the dark blocks and theirneighboring pixels in order to avoid blocking artifacts. For the rest ofpixels in non-dark blocks, only global adjustments are performed. In theinterpolation step, if a neighboring block's local mapping curve doesnot exist, the global mapping curve may be used. This will make a smoothtransition between the blocks with local adjustment and those withoutlocal adjustment, thereby making a seamless integration of the globaland the local enhancements. Since local enhancement is only performed ona small number of blocks in the image, the complexity increase due tolocal enhancement is limited.

In some embodiments, to overcome the over-enhancing problem that oftenoccurs in a local adjustment method, the final mapped value of a pixelmay be obtained by the weighted average of its global mapped value andits local mapped value for the blocks with local adjustment. From ourexperiments, the same weight for the global and the local adjustmentsgives good results. It achieves the local adaptability, whilemaintaining the global tone and natural look of an image.

The terms and expressions which have been employed in the forgoingspecification are used therein as terms of description and not oflimitation, and there is no intention in the use of such terms andexpressions of excluding equivalence of the features shown and describedor portions thereof, it being recognized that the scope of the inventionis defined and limited only by the claims which follow.

1. A method for digital image contrast enhancement, said methodcomprising: a) determining a luminance map for each of a plurality ofcontiguous image blocks; b) selecting a point within one of saidcontiguous blocks; and c) calculating a mapped value for said point byinterpolating the mapped values corresponding to said point on each ofsaid luminance maps, wherein said interpolating is performed byweighting each of said luminance map values in proportion to the inversedistance between said point and the center of the block corresponding toeach luminance map.
 2. A method as described in claim 1 wherein saidluminance maps for each of said blocks are determined by a local,block-based histogram equalization method.
 3. A method as described inclaim 1 wherein at least one of said luminance maps for said blocks isdetermined by a local, block-based histogram equalization method and atleast one of said luminance maps is determined by a global method.
 4. Amethod for digital image contrast enhancement, said method comprising:a) dividing an image into a plurality of contiguous image blocks; b)determining whether each block is a dark block; c) performing a globalcontrast enhancement method when the number of said dark blocks exceedsa threshold value; and d) performing a local, block-based contrastenhancement when the number of said dark blocks does not exceed saidthreshold value.
 5. A method as described in claim 4 wherein saiddetermining whether each block is a dark block comprises determiningwhether the majority of pixels in said block are darker than a thresholdvalue.
 6. A method as described in claim 5 wherein said threshold valueis
 64. 7. A method as described in claim 4 wherein said global contrastenhancement comprises a histogram equalization method.
 8. A method asdescribed in claim 4 wherein said local, block-based contrastenhancement comprises a histogram equalization method withcontiguous-block luminance map interpolation.
 9. A method for digitalimage contrast enhancement, said method comprising: a) dividing an imageinto a plurality of contiguous image blocks; b) determining whether eachblock is a dark block; c) determining a global luminance map for thewhole image; d) determining a local, block-based luminance map for saiddark blocks; e) calculating a final luminance map for said dark blocksand said non-dark blocks that are adjacent to said dark blocks byinterpolating between the luminance map values for contiguous blocks; f)calculating a final luminance map for said non-dark blocks that are notadjacent to said dark blocks by applying said global luminance mapdirectly; and g) applying said final luminance maps to said image.
 10. Amethod as described in claim 9 wherein said determining whether eachblock is a dark block comprises determining whether the majority ofpixels in said block are darker than a threshold value.
 11. A method asdescribed in claim 9 wherein said global luminance map is generatedusing a histogram equalization method.
 12. A method as described inclaim 9 wherein said local, block-based luminance map is generated usinga histogram equalization method with contiguous-block luminance mapaveraging.
 13. A method as described in claim 9 wherein saidinterpolating comprises weighting each of said luminance map values ininverse proportion to the inverse distance between said point and thecenter of the block corresponding to each luminance map.
 14. A method asdescribed in claim 9 further comprising calculating a final luminancemap for said dark blocks and said non-dark blocks that are adjacent tosaid dark blocks by a weighted average of the local, block-basedluminance map as applied to said dark blocks and the global luminancemap as applied to said non-dark blocks.